Information processing apparatus and information processing method

ABSTRACT

An information processing apparatus capable of executing calibration for improving accuracy of gaze detection without causing a user to feel stress. The information processing apparatus, includes a marker control unit that changes, during calibration of an eyewear terminal, a display position of a point-of-regard marker displayed on a display unit of the eyewear terminal; a computational processing unit that computes an optical axis vector expressing a gaze direction of a user by a pupil-corneal reflection method, on a basis of a captured image that includes a user&#39;s eye imaged when the eye of the user wearing the eyewear terminal is irradiated with light from a light source, and the point-of-regard marker is displayed at a calibration point; and an evaluation unit that evaluates a variation of the optical axis vector computed for a plurality of the calibration points.

CROSS-REFERENCE TO RELATED APPLICATION

The present application is a continuation application of U.S. patentapplication Ser. No. 15/531,963, filed May 31, 2017, which is a nationalstage entry of PCT/JP2015/075894, filed Sep. 11, 2015, which claimspriority from prior Japanese Priority Patent Application JP 2014-254720filed in the Japan Patent Office on Dec. 17, 2014, the entire contentsof which are hereby incorporated by reference.

TECHNICAL FIELD

The present disclosure relates to an information processing apparatus,an information processing method, and a program.

BACKGROUND ART

There is disclosed technology that detects a user's gaze with respect toa display screen displaying a variety of content, and utilizes thedetected gaze for various types of operations. For example, PatentLiterature 1 discloses an imaging apparatus that radiates light in theinfrared band (infrared light) onto the eye of a user peering into aviewfinder, and detects the user's gaze with respect to a display screendisplaying a through-the-lens image by capturing the reflected lightfrom the eye with a detector, and also utilizes the detected gaze forautofocus (AF).

CITATION LIST Patent Literature

Patent Literature 1: JP H5-333259A

DISCLOSURE OF INVENTION Technical Problem

Recently, the development of wearable terminals such as head-mounteddisplays and eyeglasses-style terminals has been advancing rapidly. Suchtechnology that detects the user's gaze is also important in a wearableterminal worn by the user to view a display, and the detected gaze isused as terminal operation information, or used as autofocusinformation, for example. Particularly, in a wearable terminal, it isdesirable to conduct calibration that improves the accuracy of gazedetection without causing the user to feel stress.

Accordingly, the present disclosure proposes a novel and improvedinformation processing apparatus, information processing method, andprogram capable of executing calibration for improving the accuracy ofgaze detection without causing the user to feel stress.

Solution to Problem

According to the present disclosure, there is provided an informationprocessing apparatus, including: a marker control unit that changes,during calibration of an eyewear terminal, a display position of apoint-of-regard marker displayed on a display unit of the eyewearterminal; a computational processing unit that computes an optical axisvector expressing a gaze direction of a user by a pupil-cornealreflection method, on a basis of a captured image that includes a user'seye imaged when the eye of the user wearing the eyewear terminal isirradiated with light from a light source, and the point-of-regardmarker is displayed at a calibration point; and an evaluation unit thatevaluates a variation of the optical axis vector computed for aplurality of the calibration points.

According to the present disclosure, there is provided an informationprocessing method, conducted by an information processing apparatus, themethod including: changing, during calibration of an eyewear terminal, adisplay position of a point-of-regard marker displayed on a display unitof the eyewear terminal; computing an optical axis vector expressing agaze direction of a user by a pupil-corneal reflection method, on abasis of a captured image that includes a user's eye imaged when the eyeof the user wearing the eyewear terminal is irradiated with light from alight source, and the point-of-regard marker is displayed at acalibration point; and evaluating a variation of the optical axis vectorcomputed for a plurality of the calibration points.

According to the present disclosure, there is provided a program causinga computer to function as an information processing apparatus including:a marker control unit that changes, during calibration of an eyewearterminal, a display position of a point-of-regard marker displayed on adisplay unit of the eyewear terminal; a computational processing unitthat computes an optical axis vector expressing a gaze direction of auser by a pupil-corneal reflection method, on a basis of a capturedimage that includes a user's eye imaged when the eye of the user wearingthe eyewear terminal is irradiated with light from a light source, andthe point-of-regard marker is displayed at a calibration point; and anevaluation unit that evaluates a variation of the optical axis vectorcomputed for a plurality of the calibration points.

Advantageous Effects of Invention

According to the present disclosure as described above, it is possibleto execute calibration for improving the accuracy of gaze detectionwithout causing the user to feel stress. Note that the effects describedabove are not necessarily limitative. With or in the place of the aboveeffects, there may be achieved any one of the effects described in thisspecification or other effects that may be grasped from thisspecification.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is an explanatory diagram illustrating the structure of an eye.

FIG. 2 is an explanatory diagram illustrating a configuration on theside facing the user's eyes in an eyewear terminal according to anembodiment of the present disclosure.

FIG. 3 is a schematic side view illustrating the positional relationshipbetween the user's eye and an eyewear terminal when an eyewear terminalaccording to the embodiment is worn.

FIG. 4 is a function block diagram illustrating a functionalconfiguration of an eyewear terminal and an information processingapparatus according to the embodiment.

FIG. 5 is a flowchart illustrating a calibration process of an eyewearterminal performed by an information processing apparatus according toan embodiment.

FIG. 6 is an explanatory diagram illustrating an example display of apoint-of-regard that is displayed moving.

FIG. 7 is an explanatory diagram for explaining a process of computingan optical axis vector using a pupil-corneal reflection method.

FIG. 8 is an explanatory diagram illustrating a relationship between amarker vector and an optical axis vector.

FIG. 9 is an explanatory diagram illustrating an example of calibrationpoints at which gaze data is acquired.

FIG. 10 is an explanatory diagram illustrating another example ofcalibration points at which gaze data is acquired.

FIG. 11 is a graph illustrating an example of evaluation results ofoptical axis variation.

FIG. 12 is an explanatory diagram illustrating coordinates of a markervector and an optical axis vector.

FIG. 13 is an explanatory diagram for explaining the modification of thepositions of calibration points.

FIG. 14 is an explanatory diagram illustrating an example of brightpoints appearing on the eye when light is radiated from a light source.

FIG. 15 is a hardware configuration diagram illustrating a hardwareconfiguration of an information processing apparatus according to theembodiment.

MODE(S) FOR CARRYING OUT THE INVENTION

Hereinafter, (a) preferred embodiment(s) of the present disclosure willbe described in detail with reference to the appended drawings. In thisspecification and the appended drawings, structural elements that havesubstantially the same function and structure are denoted with the samereference numerals, and repeated explanation of these structuralelements is omitted.

Hereinafter, the description will proceed in the following order.

-   1. Overview-   2. Hardware configuration of eyewear terminal-   3. Functional configuration-   4. Calibration process-   (1) Display point-of-regard marker (S100)-   (2) Acquire gaze data (S110 to S140)-   (3) Evaluate (S150 to S180)-   5. Adaptation to detection accuracy improvement-   5.1. Pairing with bright points-   5.2. Dynamic change of point-of-regard marker display position-   6. Hardware configuration

<1. Overview>

First, an overview of an information processing apparatus according toan embodiment of the present disclosure will be described with referenceto FIG. 1. Note that FIG. 1 is an explanatory diagram illustrating thestructure of an eye.

An information processing apparatus according to the present embodimentis an apparatus that performs calibration executed to improve gazedetection accuracy when detecting the user's gaze with respect to adisplay. In the present embodiment, the user's gaze is detected using apupil-corneal reflection method. The pupil-corneal reflection method isa technique that radiates light from a light source onto the user's eye,detects the reflected light from that light on the corneal surface andthe position of the pupil, and estimates the gaze direction.

Herein, as illustrated in FIG. 1, the user's gaze lies on a sight axisAs joining a node 12 a on the central rear face of the crystalline lens12 of the eye 10, and the fovea centralis 16 a. Meanwhile, the gazedirection estimated by the above pupil-corneal reflection method lies onan optical axis Ao on the normal line of the cornea 14 that passesthrough the center of the pupil 17. A discrepancy exists between thesight axis As and the optical axis Ao, and although dependent on theindividual, typically tends to range from approximately 4° to 8°. Asthis discrepancy becomes larger, gaze detection accuracy falls, and thuscalibration is performed to correct this discrepancy.

The calibration is conducted according to the following procedure.

-   (Step 1) Estimate the optical axis when looking at a point inside    the field of view (hereinafter also designated a “point-of-regard”)-   (Step 2) Measure the difference between the point-of-regard vector    from the corneal center of curvature to the point-of-regard, and a    vector of the estimated optical axis-   (Step 3) On the basis of the difference measured in (Step 2),    estimate the sight axis at the time from the optical axis when    looking at an arbitrary point

Note that since the eye 10 rotates by muscular pulling, a roll rotationis imparted depending on the direction in which to look. For thisreason, the calibration parameter differs depending on the orientationof the eye 10. Accordingly, parameters ordinarily are acquired atmultiple points of regard (for example, from 5 to 9 points) within thefield of view.

With such calibration, error exists in the detection of reflected lightfrom the pupil surface and the estimation of the optical axis. Bymoderating the variation of this error, it becomes possible to raise theaccuracy of gaze detection. Accordingly, in an information processingapparatus according to the present embodiment, calibration is performedto moderate the variation of error. At this point, various processes areconducted so that calibration is executed without causing the user tofeel stress. Hereinafter, the configuration and the functions of aninformation processing apparatus according to the present embodimentwill be described in detail.

<2. Hardware Configuration of Eyewear Terminal>

Prior to describing an information processing apparatus according to thepresent embodiment, a hardware configuration of an eyewear terminal 100in which the calibration by the information processing apparatusaccording to the present embodiment is conducted will be described onthe basis of FIGS. 2 and 3. Note that FIG. 2 is an explanatory diagramillustrating a configuration on the side facing the user's eyes in theeyewear terminal 100. FIG. 3 is a schematic side view illustrating thepositional relationship between the user's eye 10 and the eyewearterminal 100 when the eyewear terminal 100 is worn.

The eyewear terminal 100 is a device which is worn on the user's head,and which is used in a state of the eyes facing one or more displayunits. The eyewear terminal 100 is a device such as a head-mounteddisplay or an eyeglasses-style terminal, for example. As illustrated inFIG. 2, on the face on the side facing the user's eyes in the eyewearterminal 100 according to the present embodiment, respective displayunits 102R and 102L are provided at positions corresponding to the righteye and the left eye. The display units 102R and 102L according to thepresent embodiment are formed in an approximately rectangular shape.Note that on a housing 101, a depression 101 a where the user's nose ispositioned may also be formed between the display units 102R and 102L.

Around the perimeter of the display unit 102R, four light sources 103Ra,103Rb, 103Rc, and 103Rd are respectively provided in the approximatemiddle of the four sides of the display unit 102R. Similarly, around theperimeter of the display unit 102L, four light sources 103La, 103Lb,103Lc, and 103Ld are respectively provided in the approximate middle ofthe four sides of the display unit 102L. These light sources 103Ra to103Rd and 103La to 103Ld are made up of a light source that emitsinfrared light. The light sources 103Ra to 103Rd and 103La to 103Ldradiate light onto the user's eye 10 that faces the display unit 102R or102L around which the light sources are respectively provided.

Additionally, around the perimeter of the display units 102R and 102L,imaging unit 104R and 104L that capture an image of the eye 10 areprovided, respectively. As illustrated in FIG. 2, for example, each ofthe imaging units 104R and 104L is provided under each of the displayunits 102R and 102L (below the light sources 103Rc and 103Lc providedunder the display units 102R and 102L). As illustrated in FIG. 3, theimaging units 104R and 104L are disposed so that at least the pupil 17of the eye 10 to capture is included in the capture range. For example,the imaging units 104R and 104L are disposed having a certain elevationangle θ. The elevation angle θ may be set to approximately 30°, forexample.

Note that the eyewear terminal 100 is configured so that when worn bythe user, the display units 102R and 102L are separated from the user'seyes 10 by a certain distance. Consequently, the user wearing theeyewear terminal 100 is able to keep the display regions of the displayunits 102R and 102L within his or her field of view without discomfort.At this point, the distance between the display units 102R and 102L andthe user's eyes 10 may be decided so that even if the user is wearingglasses G, the eyewear terminal 100 is still wearable over the glassesG. In this state, the imaging units 104R and 104L are disposed so thatthe pupil 17 of the user's eye 10 is included in the capture range.

<3. Functional Configuration>

Next, a functional configuration of the eyewear terminal 100 discussedabove and an information processing apparatus 200 that performscalibration of the eyewear terminal 100 will be described on the basisof FIG. 4. Note that FIG. 4 is a function block diagram illustrating afunctional configuration of the eyewear terminal 100 and the informationprocessing apparatus 200.

[3.1. Eyewear Terminal]

As illustrated in FIG. 4, the eyewear terminal 100 is equipped with alight source 110, an imaging unit 120, a display unit 130, a controlunit 140, and a transceiving unit 150.

The light source 110 radiates light on the eye 10 of a user wearing theeyewear terminal 100. The light source 110 is a light source that emitsinfrared light, for example, and corresponds to the light sources 103Rato 103Rd and 103La to 103Ld in FIG. 2. The light source 110 emits lighton the basis of an instruction from the control unit 140.

The imaging unit 120 captures the eye 10 of the user wearing the eyewearterminal 100. The imaging unit 120 corresponds to the imaging units 104Rand 104L in FIG. 2. The imaging unit 120 conducts capture on the basisof an instruction from the control unit 140, and outputs a capturedimage to the control unit 140.

The display unit 130 is an output unit that displays information. Thedisplay unit 130 corresponds to the display units 102R and 102L in FIG.2. The display unit 130 may be a liquid crystal display or an organic ELdisplay, or alternatively, may be a lens on which information isdisplayed by a projection device, for example. Information is displayedon the display unit 130 according to an instruction from the controlunit 140.

The control unit 140 controls the functions of the eyewear terminal 100overall. The control unit 140 may perform control lighting control ofthe light source 110, perform capture control of the imaging unit 120,and display information on the display unit 130, for example.Additionally, the control unit 140 controls the transmission andreception of information to and from the information processingapparatus 200 via the transceiving unit 150.

The transceiving unit 150 is an interface that transmits and receivesinformation to and from external equipment. In the present embodiment,calibration is conducted in the eyewear terminal 100 by transmitting andreceiving information to and from the information processing apparatus200. At this point, a captured image captured by the imaging unit 120 istransmitted from the eyewear terminal 100 to the information processingapparatus 200 via the transceiving unit 150. Also, informationtransmitted from the information processing apparatus 200, such aslighting control information for the light source 110 duringcalibration, capture control information causing the imaging unit 120 toperform capture, and display information causing the display unit 130 todisplay information, is received via the transceiving unit 150.

[3.2. Information Processing Apparatus]

Next, as illustrated in FIG. 4, the information processing apparatus 200is equipped with a transceiving unit 210, a marker control unit 220, acomputational processing unit 230, a storage unit 240, and an evaluationunit 250.

The transceiving unit 210 is an interface that transmits and receivesinformation to and from external equipment. In the present embodiment,the transceiving unit 210 transmits and receives information forexecuting calibration to and from the eyewear terminal 100. At thispoint, the transceiving unit 210 transmits information, such as lightingcontrol information for the light source 110 during calibration, capturecontrol information causing the imaging unit 120 to perform capture, anddisplay information causing the display unit 130 to display information,to the eyewear terminal 100. Additionally, the transceiving unit 210receives information such as a captured image captured by the imagingunit 120 from the eyewear terminal 100.

The marker control unit 220 controls the display of a point-of-regardmarker displayed on the display unit 130 of the eyewear terminal 100during calibration. The point-of-regard marker is an object displayed inthe display region to measure the discrepancy between the user's opticalaxis and sight axis. By causing the user to direct his or her gaze atthe displayed point-of-regard marker, a vector from the user's pupilcenter to the point-of-regard marker (hereinafter also designated the“marker vector”) may be obtained, and in addition, the user's opticalaxis at that time is also estimated.

The marker control unit 220 successively displays the point-of-regardmarker at certain positions (hereinafter also designated “calibrationpoints”) so that the user's gaze data is acquired at multiple positionsinside the display region. After a certain amount of gaze data isacquired at a calibration point where the point-of-regard is displayed,the marker control unit 220 repeats the process of moving thepoint-of-regard marker to the next calibration point, and acquires theuser's gaze data at all calibration points.

At this point, the marker control unit 220 moves the point-of-regardmarker between respective calibration points while keeping thepoint-of-regard marker displayed. As a result, the user's gaze moves tofollow the point-of-regard marker, and compared to the case ofintermittently displaying the point-of-regard marker, time to search forthe point-of-regard marker displayed at a calibration point becomesunnecessary, and the movement of the gaze directed at thepoint-of-regard marker may be stabilized.

In addition, the marker control unit 220 may also control the movementspeed of the point-of-regard marker moving between calibration points.If the point-of-regard marker is moved at a constant speed, the gazetends to be less likely to settle down when the point-of-regard markeris displayed at the destination calibration point. Accordingly, themarker control unit 220 may control the movement speed of thepoint-of-regard marker moving between calibration points to slow down asthe point-of-regard marker approaches the destination calibration point.Consequently, the point-of-regard marker moves quickly immediately afterthe start of movement, but slows down as the point-of-regard markerapproaches the destination calibration point. Since the user's gazemoves along with the movement speed of the point-of-regard marker, themovement of the user's gaze also relaxes as the point-of-regard markerapproaches the destination calibration point, and the gaze settles downmore easily when the point-of-regard marker is dpisplayed at thecalibration point.

The computational processing unit 230 respectively computes the user'soptical axis and the marker vector when the point-of-regard marker isdisplayed at each calibration point. The computational processing unit230 acquires from the eyewear terminal 100 a captured image depictingthe user's eye regarding the point-of-regard marker while light from alight source is radiated onto the user's eyeball, and computes theuser's optical axis and the marker vector. The computed optical axis andmarker vector are stored in the storage unit 240 for each calibrationpoint.

The storage unit 240 stores various information required duringcalibration of the eyewear terminal 100. For example, the storage unit240 stores movement information stipulating the positions of calibrationpoints at which to display the point-of-regard marker and how thepoint-of-regard marker is to move, and settings information such as theamount of gaze data to acquire at each calibration point and a thresholdvalue used to determine the end of calibration. In addition, the storageunit 240 also stores gaze data computed by the computational processingunit 230.

The evaluation unit 250 determines the end of calibration of the eyewearterminal 100. The evaluation unit 250 determines whether or not theeyewear terminal 100 is correctly calibrated by determining whether ornot the variation in the user's optical axis estimated at eachcalibration point is inside an allowed range. This determination processwill be discussed in detail later. If the determination result from theevaluation unit 250 is that the variation in the user's optical axis isdetermined not to be contained inside a certain range, the calibrationparameters are modified, and calibration is executed again.

The above thus describes a functional configuration of the eyewearterminal 100 and the information processing apparatus 200. Note that inFIG. 4, the information processing apparatus 200 that conducts thecalibration process is illustrated as being separate from the eyewearterminal 100, but the present disclosure is not limited to such anexample. For example, some or all of the functions of the informationprocessing apparatus 200 illustrated in FIG. 3 may also be installedonboard the eyewear terminal 100.

<4. Calibration Process>

Next, the calibration process of the eyewear terminal 100 performed bythe information processing apparatus 200 according to the presentembodiment will be described on the basis of FIGS. 5 to 14. Note thatFIG. 5 is a flowchart illustrating the calibration process of theeyewear terminal 100 performed by the information processing apparatus200 according to the present embodiment. FIG. 6 is an explanatorydiagram illustrating an example display of a point-of-regard that isdisplayed moving. FIG. 7 is an explanatory diagram for explaining aprocess of computing an optical axis vector using a pupil-cornealreflection method. FIG. 8 is an explanatory diagram illustrating arelationship between a marker vector and an optical axis vector. FIG. 9is an explanatory diagram illustrating an example of calibration pointsat which gaze data is acquired. FIG. 10 is an explanatory diagramillustrating another example of calibration points at which gaze data isacquired. FIG. 11 is a graph illustrating an example of evaluationresults of optical axis variation. FIG. 12 is an explanatory diagramillustrating coordinates of a marker vector and an optical axis vector.FIG. 13 is an explanatory diagram for explaining the modification of thepositions of calibration points. FIG. 14 is an explanatory diagramillustrating an example of bright points appearing on the eye when lightis radiated from a light source.

(1) Display Point-of-Regard Marker (S100)

The calibration process of the eyewear terminal 100 performed by theinformation processing apparatus 200 according to the present embodimentstarts by displaying the point-of-regard on the display unit 130, andcausing the user to direct his or her gaze at the point-of-regard marker(S100). The display control of the point-of-regard marker is conductedby the control unit 140, which receives instructions from the markercontrol unit 220 of the information processing apparatus 200. In thecalibration, the user's gaze data is acquired at multiple positionsinside the display region of the display unit 130. By displaying thepoint-of-regard marker at a calibration point that acts as a position atwhich to acquire gaze data, the user intentionally directs his or hergaze at the point-of-regard marker, making it possible to acquire gazedata.

The point-of-regard marker is successively displayed at multiple presetcalibration points inside the display region 300 of the display unit130. As illustrated in FIG. 6, for example, the point-of-regard markerfirst is displayed at a calibration point CP1 in the center of thedisplay region 300. When the point-of-regard marker M is displayed atthe calibration point CP1, the user directs his or her gaze at thepoint-of-regard marker M. By keeping the point-of-regard marker Mdisplayed, the user's gaze may be fixed on the calibration point CP1,and gaze data is acquired in this state.

After gaze data is acquired at the calibration point CP1, thepoint-of-regard marker M, while still being displayed, is moved to thenext position at which to acquire gaze data, namely the calibrationpoint CP2 in the upper-left of the display region 300. Subsequently,gaze data at the calibration point CP2 is acquired. After that, theacquisition of gaze data and the movement are repeated for thecalibration point CP3 at the upper-right of the display region 300, atthe calibration point CP4 at the lower-left of the display region 300,and the calibration point CP5 at the lower-right of the display region300. In step S100, the point-of-regard marker M is displayed at thefirst calibration point, and the calibration process is started.

(2) Acquire Gaze Data (S110 to S140)

After the point-of-regard marker M is displayed at the first calibrationpoint in step S100, the user's gaze data at that calibration point isacquired (S110). The gaze data includes an optical axis vectorexpressing the estimated gaze direction of the user, and a marker vectorfrom the user's pupil center to the point-of-regard marker.

(Computation of Optical Axis Vector)

The optical axis vector is estimated using the pupil-corneal reflectionmethod, for example. A process of estimating the optical axis using thepupil-corneal reflection method will now be described on the basis ofFIG. 7. In the pupil-corneal reflection method, light from a lightsource 21 is radiated onto the eye 10 of the user observing the displayscreen 23 of the display unit, and the eye 10 irradiated by the light isimaged by an imaging unit 22. Subsequently, the optical axis isestimated on the basis of a captured image 30 imaged by the imaging unit22. For the sake of simplicity, the case of irradiating the eye 10 withone light source 21 will be described herein.

As illustrated in FIG. 7, suppose that the user is regarding thepoint-of-regard marker M displayed on the display screen 23. At thispoint, the eye 10 is irradiated with light from the light source 21, andthe eye 10 is imaged by the imaging unit 22. As illustrated in FIG. 7,the acquired captured image 30 of the eye 10 depicts the cornea 14, theiris 13, and the pupil 17 of the user's eye 10. The captured image 30also depicts a Purkinje image P, which is a bright point of irradiatinglight irradiating the eye 10 from the light source 21.

After the captured image 30 is acquired, a process of computing theoptical axis is conducted. The process of computing the optical axis isconducted by the computational processing unit 230. For this reason,first, the pupil center S and the Purkinje image P are detected from thecaptured image 30. These detection processes may be conducted by knownimage recognition technology.

For example, in the process of detecting the image of the pupil 17,various types of image processing with respect to the captured image 30(for example, processing to adjust factors such as distortion, blacklevel, and white balance), a process of acquiring a luminancedistribution inside the captured image 30, and the like are conducted.Also, on the basis of the acquired luminance distribution, a process ofdetecting an edge in the image of the pupil 17, a process ofapproximating the detected edge in the image of the pupil 17 with afigure such as a circle or an ellipse, and the like may be conducted.From the detected image of the pupil 17, the pupil center S may becomputed.

Additionally, in the process of detecting the Purkinje image P, a seriesof processes, such as the various types of image processing with respectto the captured image 30, the process of acquiring a luminancedistribution inside the captured image 30, and a process of detectingpixels where the difference in the luminance value is comparativelylarger than the surrounding pixels on the basis of the luminancedistribution, may also be conducted. Also, the center of the Purkinjeimage P may be detected from the detected Purkinje image P.

Next, three-dimensional coordinates of the pupil center S and acurvature center C of the cornea 14 are computed. When the cornea 14 istaken to be part of a sphere, the curvature center C of the cornea 14 isthe center of the sphere. The three-dimensional coordinates of the pupilcenter S are computed on the basis of the image of the pupil 17 detectedfrom the captured image 30. Specifically, the three-dimensionalcoordinates of each point on the edge of the pupil 17 in the capturedimage 30 are computed on the basis of properties such as the positionalrelationship between the imaging unit 22 and the eye 10, the refractionof light at the surface of the cornea 14, and the distance between thecurvature center C of the cornea 14 and the pupil center S. The centerpoint of these coordinates is taken to be the three-dimensionalcoordinates of the pupil center S.

In addition, the curvature center C of the cornea 14 is computed on thebasis of the Purkinje image P and the center thereof detected from thecaptured image 30. Specifically, on a straight line joining the imagingunit 22 and the center of the Purkinje image P, the position advancedfrom the surface of the cornea 14 inward into the eye 10 by thecurvature radius of the cornea 14 is computed as the three-dimensionalcoordinates of the curvature center C of the cornea 14, on the basis ofproperties such as the positional relationship between the light source21, the imaging unit 22, and the eye 10, and the curvature radius of thecornea 14.

The straight line joining the curvature center C of the cornea 14computed in this way and the pupil center S becomes the estimatedoptical axis. In other words, the coordinates of the position at whichthe optical axis and the display screen 23 intersect become theestimated gaze position of the user. Note that a vector proceeding fromthe curvature center C of the cornea 14 to the pupil center S isdesignated the optical axis vector vo.

(Computation of Marker Vector)

Meanwhile, the marker vector from the user's pupil center S to thepoint-of-regard marker M may be computed as the vector proceeding fromthe pupil center S specified from the captured image 30 as discussedabove to the position on the display screen 23 where the point-of-regardmarker M is displayed currently.

In this way, in step S110, the optical axis vector and the marker vectorare computed and acquired by the computational processing unit 230 asgaze data. The acquired gaze data is stored in the storage unit 240.

(Moderating Variation in Detection Results)

At this point, the computational processing unit 230 may also determinewhether or not the computed optical axis vector vo is usable as acalibration detection result.

Specifically, for example, the computational processing unit 230 maydetermine whether or not wobbling of the optical axis vector vo iswithin a certain range, and confirm that the computed optical axisvector vo is not greatly divergent from the optical axis vectors voacquired so far. Optical axis vectors vo computed by the computationalprocessing unit 230 are stored as a history in the storage unit 240.Using this history, the computational processing unit 230 confirms thatthe angle obtained between an average vo_ave of the previous N acquiredoptical axis vectors, including the current computation, and the currentoptical axis vector vo is inside a certain value, for example.Additionally, when the angle obtained between the optical axis vectoraverage vo_ave and the current optical axis vector vo exceeds a certainthreshold value, the currently computed optical axis vector vo istreated as a large wobble, and is not used as a calibration detectionresult. As a result, the accuracy of the optical axis vector may beincreased.

The optical axis vector average vo_ave may be computed using theprevious three optical axis vectors vo, for example. Also, the thresholdvalue for determining the angle obtained between the optical axis vectoraverage vo_ave and the current optical axis vector vo may be set toapproximately 3°, for example. This threshold value is decided whileaccounting for the inconsistency of detection. According to such adetermination, when given the estimated user gaze positions m1, m2, andm3 as illustrated in FIG. 8, for example, a point like the gaze positionm3 which is divergent from the others may be excluded from the detectionresults. Additionally, even if the optical axis vector vo is computedfrom a captured image that is imaged when the user is not looking at thepoint-of-regard marker M, the computed optical axis vector vo becomesgreatly divergent from the optical axis vector average vo_ave. Theoptical axis vector vo in such a case may also be excluded from thedetection results by the determination.

In addition, the computational processing unit 230 may also determinewhether or not an angle w obtained between the computed marker vector vmand the optical axis vector vo is less than or equal to a certain value,for example. According to such a determination, it is possible toconfirm whether the estimated optical axis vector vo is greatlydivergent from the actual gaze direction. The value of the thresholdused at this point is decided with consideration for factors such as thediscrepancy between the optical axis and the sight axis, and the opticalaxis detection error.

For example, the estimated gaze direction of the user (in other words,the optical axis) and the direction in which the user actually islooking (in other words, the sight axis) do not necessarily match. Thisis due to factors such as the shape and size of the eyeball, and thearrangement of the retina and optic nerve in the eyeball. Although thereis individual variation, the discrepancy between the optical axis andthe sight axis ordinarily is from 4° to 8°. Also, an optical axisdetection error of several degrees, such as up to ±3°, for example, isconsidered to exist. If this error is combined with other accumulativeerror of ±1°, an error ranging approximately from 0° to 12° isanticipated to occur. In this case, if the angle w obtained between thecomputed marker vector and the optical axis vector is inside the rangefrom 0° to 12°, the computed optical axis vector vo may be treated asbeing of allowable accuracy, and may be used as a calibration detectionresult.

By conducting such a determination process, variation in the detectionresults may be moderated, and the accuracy of the optical axis vectormay be increased.

(Misdetection Determination)

Furthermore, even if detection clears the above determination formoderating variation in the detection results, the pupil or the brightpoint may be continually detected at an incorrect location in somecases. If an incorrect detection result is used, the calibration processcannot be conducted correctly. Accordingly, the computational processingunit 230 may also conduct a misdetection determination process so as notto use such an incorrect detection result as a calibration detectionresult. For example, if the computed sizes of the left and right pupilsare extremely different, there is a strong possibility that an incorrectlocation may be recognized as a pupil. The gaze data acquired in such acase is not used as a detection result. As a specific example, if thesize ratio of the left and right pupils exceeds a certain value (forexample, 1.2), the sizes of the left and right pupils may be treated asbeing extremely different, and the acquired gaze data may not be used asa detection result.

After the above processes are conducted, the computational processingunit 230 determines whether or not gaze data has been acquired at thecalibration point where the point-of-regard marker M is displayedcurrently (S120). For example, if a correct result has not beenobtained, such as when wobbling is determined to exist in the opticalaxis vector vo from past data, or when the angle w obtained between themarker vector vm and the optical axis vector vo is not inside theallowed range, the eye 10 is imaged and gaze data is acquired again.

Meanwhile, when gaze data is acquired at the calibration point where thepoint-of-regard marker M is displayed currently, the computationalprocessing unit 230 determines whether or not gaze data has beenacquired for all calibration points (S130). The calibration points atwhich to acquire gaze data are stored in advance in the storage unit240. If there exists a calibration point at which gaze data has not beenacquired, the computational processing unit 230 instructs the markercontrol unit 220 to move the point-of-regard marker M to the nextcalibration point (S140). The marker control unit 220 outputs, to theeyewear terminal 100 via the transceiving unit 210, an instruction tomove the point-of-regard marker M to the next preset calibration point.

(Point-of-Regard Marker Movement Process)

The point-of-regard marker M is displayed in order to direct the user'sgaze. Herein, the display of the point-of-regard marker M is controlledto enable correct acquisition of the user's gaze data in a short time.

First, the point-of-regard marker M moves between respective calibrationpoints while remaining displayed. As a result, the user's gaze moves tofollow the point-of-regard marker, and compared to the case ofintermittently displaying the point-of-regard marker M, time to searchfor the point-of-regard marker M displayed at a calibration pointbecomes unnecessary, and the movement of the gaze directed at thepoint-of-regard marker may be stabilized.

Additionally, the movement speed of the point-of-regard marker M movingbetween the calibration points is varied. If the point-of-regard markerM is moved at a constant speed, there is a tendency for the gaze to beless likely to settle when the point-of-regard marker M is displayed atthe destination calibration point. Accordingly, the marker control unit220 controls the movement speed of the point-of-regard marker M movingbetween calibration points to slow down as the point-of-regard marker Mapproaches the destination calibration point. Consequently, thepoint-of-regard marker M moves quickly immediately after the start ofmovement, but slows down as the point-of-regard marker M approaches thedestination calibration point. Since the user's gaze moves along withthe movement speed of the point-of-regard marker, the movement of theuser's gaze also relaxes as the point-of-regard marker M approaches thedestination calibration point, and the gaze settles down more easilywhen the point-of-regard marker M is displayed at the calibration point.

In addition, the calibration points at which to acquire gaze data in thedisplay region 300 ordinarily are set in the center of the displayregion 300, which is the position that the user looks at when facingforward, and near the periphery of the display region 300, which iswhere the discrepancy between the sight axis and the optical axis tendsto be larger. Ordinarily, multiple points (for example, from 5 to 9points) inside the field of view are set as calibration points. Byconducting calibration at these positions, a correction process may beconducted so that the appearance of the display region 300 becomesuniform overall. Specifically, as illustrated in FIG. 9, for example,calibration may be conducted in the center (calibration point CP1) andthe four corners (calibration points CP2 to CP5) of a rectangulardisplay region 300. Alternatively, as illustrated in FIG. 10,calibration may be conducted in the center (calibration point CP1) andnear the midpoint of each edge (calibration points CP2 to CP5) of arectangular display region 300.

At this point, when moving the point-of-regard marker M betweenrespective calibration points, the movement sequence of thepoint-of-regard marker M may be decided so that the movement distance isas large as possible. The user moves his or her gaze along with themovement of the point-of-regard marker M, but if the movement distanceof the point-of-regard marker M is small, the gaze is less likely toalign with the point-of-regard marker M displayed at the nextcalibration point, and the discrepancy between the sight axis and theoptical axis becomes larger. Also, since the discrepancy between thesight axis and the optical axis also tends to become larger when thepoint-of-regard marker M is moved in the horizontal direction of thedisplay region 300, the point-of-regard marker M may also be moved so asto include movement in the vertical direction, such as moving up anddown or diagonally.

For example, when acquiring gaze data at the five calibration points CP1to CP5 set in the center and the four corners of the display region 300illustrated in FIG. 9, after displaying the calibration point CP1 in thecenter, the calibration points CP2 to CP5 in the four corners may bemoved to in a zigzag manner. Additionally, when acquiring gaze data atthe five calibration points CP1 to CP5 set in the center and near themidpoint of each edge of the display region 300 illustrated in FIG. 10,for example, first, the calibration points CP1 to CP4 near the midpointof each edge are displayed in a sequence so as to draw a diamond-shapedtrail. After that, the calibration point CP1 in the center may bedisplayed.

Returning to the description of FIG. 5, after the point-of-regard markerM is moved to the next calibration point in step S140, the acquisitionof gaze data at the destination calibration point is conducted (S110).After that, the processes from steps S110 to S140 are repeatedlyexecuted until gaze data acquisition is completed at all calibrationpoints.

(3) Evaluate (S150 to S180)

After gaze data is acquired at all calibration points, a calibrationcompletion determination is made by the evaluation unit 250. In thepresent embodiment, the calibration completion determination is made bydetermining whether or not the overall variation in the estimatedoptical axis vectors vo is inside an allowed range.

After calibration is conducted correctly, the optical axis vectors vo atthe respective calibration points computed in step S110 become valuescorresponding to the display positions of the calibration points in thedisplay region 300. Herein, FIG. 11 illustrates an example of detectionresults for the optical axis vector vo when conducting calibration withthe calibration points CP1 to CP5 illustrated in FIG. 9. FIG. 11illustrates the relationship between the angle θ in the verticaldirection of the optical axis vector vo, the angle w in the horizontaldirection of the optical axis vector vo. Note that in the presentembodiment, the optical axis vector vo is defined on the basis of thecoordinate axes illustrated in FIG. 12. In the coordinate axes of FIG.12, the x-axis represents the horizontal direction of the display region300, the y-axis represents the vertical direction of the display region300, and the z-axis represents the depth direction of the display region300. The angle θ is the angle obtained between the optical axis vectorvo and the zx plane, while the angle ω is the angle obtained between theoptical axis vector vo and the xy plane.

The upper side of FIG. 11 illustrates a distribution of optical axisvectors vo when calibration is conducted correctly, while the lower sideof FIG. 11 illustrates a distribution of optical axis vectors vo whencalibration is not conducted correctly. According to the upper side ofFIG. 11, when calibration is conducted correctly, the optical axisvectors vo are distributed with clean divisions in correspondence witheach of the calibration points set in the center and the four corners ofthe display region 300.

On the other hand, as illustrated on the lower side of FIG. 11, whencalibration is not conducted correctly, the optical axis vectors vo arenot distributed cleanly. For example, the angle θ in the verticaldirection of the optical axis vectors vo corresponding to thecalibration points in the upper-right, the upper-left, and the center ofthe display region 300 become approximately the same. In particular,such a distribution occurs easily for the wearers of hard contact lensesor users with squinted, narrow eyes.

Accordingly, in the present embodiment, the evaluation unit 250calculates a correlation relationship between the marker vector vm andthe optical axis vector vo as an evaluation value for evaluating theoverall variation in the optical axis vectors vo (S150), and makes thecalibration completion determination from the value of the correlationrelationship. A correlation coefficient r_(xy) between the marker vectorvm and the optical axis vector vo may be computed according to Formula(1) below, for example.

$\begin{matrix}{\left\lbrack {{Math}.\mspace{14mu} 1} \right\rbrack \mspace{641mu}} & \; \\{r_{xy} = \frac{\sum\limits_{i = 1}^{n}{\left( {x_{i} - \overset{\_}{x}} \right)\left( {y_{i} - \overset{\_}{y}} \right)}}{\sqrt{\sum\limits_{i = 1}^{n}\left( {x_{i} - \overset{\_}{x}} \right)^{2}}\sqrt{\sum\limits_{i = 1}^{n}\left( {y_{i} - \overset{\_}{y}} \right)^{2}}}} & (1)\end{matrix}$

Note that i is a number assigned to each calibration point, and takes avalue from 1 to n. When five calibration points are set, n is 5. Also,x_(i) and y_(i) are the x-coordinate and y-coordinate of the opticalaxis vector vo, while x⁻ and y⁻ are the x-coordinate and y-coordinate ofthe marker vector vm. Note that x⁻ and y⁻ herein denote that ⁻ isappended above x and y.

In Formula (1) above, the difference between the angle θ in the verticaldirection and the angle w in the horizontal direction for the markervector vm and the optical axis vector vo at all calibration points isevaluated. If the marker vector vm and the optical axis vector vo do notmatch at one or multiple calibration points, and the discrepancy betweenthese angles becomes large, the correlation coefficient r_(xy) computedby Formula (1) becomes smaller. The evaluation unit 250 makes thecalibration completion determination by using the correlationcoefficient r_(xy) expressing such a correlation relationship betweenthe marker vector vm and the optical axis vector vo (S160).

The calibration completion determination may also be made in accordancewith whether or not the correlation coefficient r_(xy) between themarker vector vm and the optical axis vector vo computed in step S150has fallen below a certain threshold r_(th). The threshold r_(th) may beset to 0.90, for example. In step S160, if the correlation coefficientr_(xy) between the marker vector vm and the optical axis vector vo isgreater than or equal to the threshold r^(th), the evaluation unit 250treats the overall variation of the optical axis vector vo as beinginside the allowed range, and completes the calibration process (S170).

On the other hand, if the correlation coefficient r_(xy) between themarker vector vm and the optical axis vector vo falls below thethreshold r^(th), the calibration settings information is modified tochange how calibration is conducted (S180), and calibration is performedagain. The calibration settings information refers to information suchas the display position of the point-of-regard marker M, for example.For example, the calibration point settings may be modified to shift thedisplay position of the point-of-regard marker M towards the center ofthe display region 300 or the like, and calibration may be executedagain.

For example, as illustrated in FIG. 13, suppose that calibration pointsare set with respect to the display region 300 on the basis of a regionobtained by reducing the display region 300 by a certain ratio a. Atthis point, suppose that the default values for the positions of thecalibration points are the center of the display region 300, and thefour corners of a region 90% the size of the display region, forexample. When calibration is executed by setting the calibration pointsto the default positions, and the correlation coefficient r_(xy) betweenthe marker vector vm and the optical axis vector vo falls below thethreshold r^(th), the positions of the calibration points are shiftedtowards the center of the display region. For example, the positions ofthe calibration points in the four corners are set to the four cornersof a region 80% the size of the display region. By shifting thepositions of the calibration points towards the center of the displayregion in this way, the user becomes able to see the point-of-regardmarker more easily, and correct gaze data may be acquired more easily.

The above thus describes the calibration process of the eyewear terminal100 performed by the information processing apparatus 200 according tothe present embodiment. According to the present embodiment, whenacquiring gaze data at multiple calibration points, the point-of-regardmarker M displayed to direct the user's gaze is moved to eachcalibration point while remaining displayed. At this time, the movementspeed of the point-of-regard marker M may be slowed down as thepoint-of-regard marker M approaches the position of the next calibrationpoint, thereby enabling the user's gaze to track the point-of-regardmarker M accurately.

In addition, in the calibration process according to the presentembodiment, when acquiring the optical axis vector vo at eachcalibration point, the presence or absence of divergence in the opticalaxis on the basis of previously acquired optical axis vectors may bedetermined, and the discrepancy between the marker vector vm and theoptical axis vector vo may be determined. As a result, it is possiblenot to use gaze data acquired when the user is not looking at thepoint-of-regard marker M as a calibration detection result, and therebynot lower the accuracy of the gaze detection process.

Furthermore, in the calibration process according to the presentembodiment, a correlation coefficient between the marker vector vm andthe optical axis vector vo is computed from the optical axis vectors voacquired at each calibration point as an evaluation value for evaluatingthe overall variation of the optical axis vector vo. By determiningwhether or not this relation coefficient is greater than or equal to acertain threshold value, it is possible to evaluate whether or not thegaze detection process may be conducted accurately inside the field ofview as a whole. Consequently, no matter which position in the displayregion the user looks at, the accuracy of the detected optical axisvectors vo may be maintained consistently.

According to such a process, the calibration process may be completedsimply by the user directing his or her gaze at the point-of-regardmarker M. The point-of-regard marker M is displayed and moved to enablethe user to direct his or her gaze easily. In addition, even if theoptical axis vector vo is not acquired, the calibration points areadjusted automatically so that the optical axis vector vo may beacquired, and thus the calibration process may be completed withoutcausing the user to feel stress.

<5. Adaptation to Detection Accuracy Improvement>

In the calibration process, variation in the detected optical axisvectors vo tends to occur in cases such as when the user is wearing hardcontact lenses or when the user has narrow eyes. This is because whenthe user is wearing hard contact lenses, the pupil may be distorted, anumber of bright points greater than the number of light sources may bedetected on the eyeball due to light radiated from the light sources, orthe contact lens may move over the cornea. Also, when the user hasnarrow eyes, the pupil specified from the captured image may beincomplete, or a number of bright points equal to the number of lightsources which should be detected may not be detected. Accordingly, todetect the user's optical axis vector vo correctly, a process like thefollowing may be conducted additionally.

[5.1. Pairing with Bright Points]

To detect the user's optical axis vector vo correctly, for example, aprocess of pairing bright points specified from the captured image maybe conducted, for example. In the eyewear terminal 100 according to thepresent embodiment, as illustrated in FIG. 2, four light sources 103Rato 103Rd and 103La to 103Ld are provided around the display units 102Rand 102L, respectively. When the left and right eyes are respectivelyirradiated with light from these light sources 103Ra to 103Rd and 103Lato 103Ld, the four bright points Pa, Pb, Pc, and Pd appear respectivelyin each eye, as illustrated in FIG. 14.

If the light sources 103Ra to 103Rd and 103La to 103Ld are arranged asillustrated in FIG. 2, the bright points Pa, Pb, Pc, and Pdcorresponding to the arrangement are detected in each eye 10 by thelight emitted from these light sources. However, as discussed earlier,when the user is wearing hard contact lenses or when the user has narroweyes, more bright points than the four bright points Pa, Pb, Pc, and Pdmay be detected, or the four bright points may not be detected.

Accordingly, the bright points Pa and Pc facing each other vertically,and the bright points Pb and Pd facing each other horizontally aretreated as respective pairs. Subsequently, on the basis of thepositional relationship of the light sources corresponding to the pairedbright points, when one bright point is detected, it becomes possible toestimate the position of the other bright point. For example, if thebright point Pa is detected, even if the bright point Pc is notdetected, it is possible to estimate that the bright point Pc exists ata certain position below the bright point Pa. Also, even if more brightpoints than the number of light sources are detected, on the basis ofthe positional relationship of the light sources corresponding to thebright points, it is possible to specify the paired bright points fromamong many detected bright points.

In this way, by setting pairs of bright points from the arrangement ofthe light sources, it becomes possible to estimate the approximatepositions of the bright points even if a bright point is not correctlydetected from the captured image, and the detection accuracy of theoptical axis vector vo may be improved.

[5.2. Dynamic Change of Point-of-Regard Marker Display Position]

In addition, to detect the user's optical axis vector vo correctly, thepoint-of-regard marker M may also be moved dynamically. When thepoint-of-regard marker M is displayed at a position in the displayregion 300 that is difficult for the user to direct his or her gaze at,it is anticipated that the detected optical axis vector vo will be moregreatly divergent from the marker vector vm. In such cases, even if thepoint-of-regard marker M continues to be displayed at the samecalibration point, the discrepancy between the optical axis vector voand the marker vector vm will not become smaller.

Accordingly, if the optical axis vector vo at such a calibration pointcannot be acquired within a certain amount of time, for example, thedisplay position of the point-of-regard marker M is shifted towards thecenter of the display region 300, and the process of acquiring theoptical axis vector vo is executed again. The time at which to move thepoint-of-regard marker M may be a time such as after the elapse of a fewseconds (for example, 3 seconds) from when the point-of-regard marker Mis displayed at the calibration point, for example.

The movement of the point-of-regard marker M may approach the center ofthe display region 300 by a certain ratio with respect to the distancefrom the center of the display region 300 to the current calibrationpoint, for example. Alternatively, the movement of the point-of-regardmarker M may approach the middle of the display region 300 in thehorizontal direction by a certain ratio with respect to the horizontaldistance from the horizontal middle of the display region 300 to thecurrent calibration point, for example. The certain ratio of approach bythe point-of-regard marker M may be set to approximately 10%, forexample. Consequently, the distance over which the user moves his or hergaze from a state of facing forward becomes smaller, thereby making iteasier for the user to direct his or her at the point-of-regard markerM. Thus, a smaller discrepancy between the optical axis vector vo andthe marker vector vm may be expected.

The movement of the point-of-regard marker M may also be performed untilthe optical axis vector vo is acquired, for example. For example, if theacquisition of the optical axis vector vo is conducted for a certaintime, but the optical axis vector vo is not acquired within that time,the process may be repeated by which the point-of-regard marker M isadditionally moved by the certain ratio, and the acquisition of theoptical axis vector vo is conducted again. Subsequently, when theoptical axis vector vo is acquired, for example, during the acquisitionof the optical axis vector vo at subsequent calibration points, thepoint-of-regard marker M may be displayed at the positions obtained bymoving the positions of the calibration points by the certain ratio usedwhen acquiring the current optical axis vector vo. Obviously, during theacquisition of the optical axis vector vo at subsequent calibrationpoints, the point-of-regard marker M may also be displayed at thedefault positions of the calibration points.

In this way, when the acquisition of the optical axis vector vo isunsuccessful, by dynamically moving the point-of-regard marker M to aposition where the optical axis vector vo is acquired correctly, thecorrect optical axis vector vo may be acquired.

<6. Hardware Configuration Example>

Finally, an exemplary hardware configuration of the informationprocessing apparatus 200 according to the present embodiment will bedescribed. FIG. 15 is a hardware block diagram illustrating an exemplaryhardware configuration of the information processing apparatus 200according to the present embodiment.

As described above, the information processing apparatus 200 accordingto the embodiments can be implemented as a processing device such as acomputer. As illustrated in FIG. 15, the information processingapparatus 200 includes a central processing unit (CPU) 901, read onlymemory (ROM) 902, random access memory (RAM) 903, and a host bus 904 a.Furthermore, the information processing apparatus 200 includes a bridge904, an external bus 904 b, an interface 905, an input device 906, anoutput device 907, a storage device 908, a drive 909, a connection port911, and a communication device 913.

The CPU 901 functions as an arithmetic processing unit and a controller,and controls the overall operation in the information processingapparatus 200 according to various programs. Furthermore, the CPU 901may be a microprocessor. The ROM 902 stores programs, operationparameters, and the like that the CPU 901 uses. The RAM 903 temporarilystores programs used in the execution of the CPU 901 and the parametersand the like that appropriately changes during the execution. The aboveare interconnected via a host bus 904 a constituted by a CPU bus.

The host bus 904 a is connected to the external bus 904 b, such as aperipheral component interconnect/interface (PCI) bus, through thebridge 904. Note that the host bus 904 a, the bridge 904, and theexternal bus 904 b are not necessarily configured as separate componentsbut the functions thereof may be implemented in a single bus.

The input device 906 includes input devices for the user to inputinformation, such as a mouse, a keyboard, a touch panel, a button, amicrophone, a switch, and a lever, and an input control circuit thatgenerates an input signal on the basis of the input performed by theuser and that outputs the input signal to the CPU 901. The output device907 includes, for example, a display device, such as a liquid crystaldisplay (LCD) device, an organic light emitting diode (OLED) device, ora lamp, and speech output device, such as a speaker.

The storage device 908 is an example of the storage unit of theinformation processing apparatus 200 and is a device for storing data.The storage device 908 may include a recording medium, a recordingdevice that records data in the recording medium, a readout device thatreads out data from the recording medium, and a deletion device thatdeletes data recoded in the recording medium. The storage device 908drives the hard disk and stores therein programs that the CPU 901executes and various kinds of data.

The drive 909 is a reader/writer for a recording medium and is built-inthe information processing apparatus 200 or is externally attached. Thedriver 909 reads out information recorded in a magnetic disk, an opticaldisk, or a magneto-optical disc that is mounted thereto or a removablestorage medium such as a semiconductor memory and outputs theinformation to the RAM 903.

The connection port 911 is an interface connected to an external deviceand is a port for connecting an external device that is capable of datatransmission through, for example, a universal serial bus (USB).Furthermore, the communication device 913 is a communication interfaceconstituted by, for example, a communication device or the like forconnecting to a communication network. Furthermore, the communicationdevice 913 may be a communication device corresponding to a local areanetwork (LAN), a communication device corresponding to a wireless USB,or a wired communication device that communicates through wire.

The preferred embodiment(s) of the present disclosure has/have beendescribed above with reference to the accompanying drawings, whilst thepresent disclosure is not limited to the above examples. A personskilled in the art may find various alterations and modifications withinthe scope of the appended claims, and it should be understood that theywill naturally come under the technical scope of the present disclosure.

For example, in the foregoing embodiment, if the correlation coefficientr_(xy) between the marker vector vm and the optical axis vector vo fallsbelow the threshold r^(th), the method of calibration is changed andcalibration is performed again, but the present technology is notlimited to such an example. For example, before changing the method ofcalibration, the threshold r^(th) may be lowered, and the calibrationcompletion determination may be made again. The changed threshold r^(th)may be a value obtained by lowering the previous threshold r^(th) by afixed value, or a value between the correlation coefficient r_(xy) andthe previous threshold r^(th), for example. Additionally, calibrationmay be ended, and calibration may be conducted again using calibrationinformation from when the threshold r^(th) was highest previously.

Further, the effects described in this specification are merelyillustrative or exemplified effects, and are not limitative. That is,with or in the place of the above effects, the technology according tothe present disclosure may achieve other effects that are clear to thoseskilled in the art based on the description of this specification.Additionally, the present technology may also be configured as below.

(1)

An information processing apparatus, including:

a marker control unit that changes, during calibration of an eyewearterminal, a display position of a point-of-regard marker displayed on adisplay unit of the eyewear terminal;

a computational processing unit that computes an optical axis vectorexpressing a gaze direction of a user by a pupil-corneal reflectionmethod, on a basis of a captured image that includes a user's eye imagedwhen the eye of the user wearing the eyewear terminal is irradiated withlight from a light source, and the point-of-regard marker is displayedat a calibration point; and

an evaluation unit that evaluates a variation of the optical axis vectorcomputed for a plurality of the calibration points.

(2)

The information processing apparatus according to (1), wherein

the marker control unit moves the point-of-regard marker to thecalibration points in a preset sequence while keeping thepoint-of-regard marker displayed.

(3)

The information processing apparatus according to (2), wherein

the marker control unit moves the point-of-regard marker so that amovement speed slows down as the point-of-regard marker approaches adestination calibration point.

(4)

The information processing apparatus according to any one of (1) to (3),wherein

the computational processing unit determines whether an angle obtainedbetween a currently computed optical axis vector and an average of theoptical axis vector computed on a basis of a history of optical axisvectors is greater than a certain angle, and

if the obtained angle is greater than the certain angle, thecomputational processing unit does not adopt the currently computedoptical axis vector.

(5)

The information processing apparatus according to any one of (1) to (4),wherein

the computational processing unit determines a variation of a currentlycomputed optical axis vector on a basis of whether an angle obtainedbetween the currently computed optical axis vector and a marker vectorfrom a pupil center of the user to the calibration point where thepoint-of-regard marker is displayed is less than or equal to a certainangle, and

if the obtained angle is greater than the certain angle, thecomputational processing unit does not adopt the currently computedoptical axis vector.

(6)

The information processing apparatus according to (5), wherein

the computational processing unit does not adopt the computed opticalaxis vector when a size ratio of left and right pupils of the user isgreater than or equal to a certain value.

(7)

The information processing apparatus according to any one of (4) to (6),wherein

when the optical axis vector computed by the computational processingunit is not adopted, the marker control unit moves the display positionof the point-of-regard marker towards a center of a display region.

(8)

The information processing apparatus according to any one of (1) to (7),wherein

the evaluation unit determines the variation of the optical axis vectorcomputed at all of the calibration points, on a basis of a correlationrelationship between a marker vector from a pupil center of the user tothe calibration point where the point-of-regard marker is displayed andthe computed optical axis vector.

(9)

The information processing apparatus according to (8), wherein

when the evaluation unit determines that a correlation coefficientexpressing the correlation relationship between the optical axis vectorand the marker vector has fallen below a certain threshold,

the marker control unit sets the calibration point to a position shiftedtowards a center of a display region, and executes calibration again.

(10)

The information processing apparatus according to any one of (1) to (9),wherein

the computational processing unit detects bright points due to lightradiated from a plurality of paired light sources.

(11)

An information processing method, conducted by an information processingapparatus, the method including:

changing, during calibration of an eyewear terminal, a display positionof a point-of-regard marker displayed on a display unit of the eyewearterminal;

computing an optical axis vector expressing a gaze direction of a userby a pupil-corneal reflection method, on a basis of a captured imagethat includes a user's eye imaged when the eye of the user wearing theeyewear terminal is irradiated with light from a light source, and thepoint-of-regard marker is displayed at a calibration point; and

evaluating a variation of the optical axis vector computed for aplurality of the calibration points.

(12)

A program causing a computer to function as an information processingapparatus including:

a marker control unit that changes, during calibration of an eyewearterminal, a display position of a point-of-regard marker displayed on adisplay unit of the eyewear terminal;

a computational processing unit that computes an optical axis vectorexpressing a gaze direction of a user by a pupil-corneal reflectionmethod, on a basis of a captured image that includes a user's eye imagedwhen the eye of the user wearing the eyewear terminal is irradiated withlight from a light source, and the point-of-regard marker is displayedat a calibration point; and

an evaluation unit that evaluates a variation of the optical axis vectorcomputed for a plurality of the calibration points.

REFERENCE SIGNS LIST

-   10 eye-   14 cornea-   17 pupil-   100 eyewear terminal-   110 light source-   120 imaging unit-   130 display unit-   140 control unit-   150 transceiving unit-   200 information processing apparatus-   210 transceiving unit-   220 marker control unit-   230 computational processing unit-   240 storage unit-   250 evaluation unit-   300 display region

1. An information processing apparatus, comprising: circuitry configured to: control a display screen of a mobile terminal to display a marker image at a first position in a first corner region of the display screen; control a light source of the mobile terminal to irradiate an eye of a user with infrared light; control an imaging unit of the mobile terminal to acquire a first captured image of the eye of the user while the light source irradiates the eye of the user and the marker image is displayed at the first position; determine whether first acquisition of an optical axis vector, which corresponds to a gaze direction of the user by a pupil-corneal reflection method, succeeds based on the first captured image; control, based on the determination that the first acquisition of the optical axis vector fails based on the first captured image, the display screen to move the marker image from the first position to a second position in the first corner region, wherein the second position is closer to a center of the display screen than the first position; control the imaging unit to acquire a second captured image of the eye of the user while the light source irradiates the eye of the user and the marker image is displayed at the second position; determine whether second acquisition of the optical axis vector succeeds based on the second captured image; and calibrate gaze detection, which is performed by the mobile terminal, based on the determination that the second acquisition of the optical axis vector succeeds based on the second captured image.
 2. The information processing apparatus according to claim 1, wherein the circuitry is configured to: control, based on the determination that the first acquisition of the optical axis vector succeeds based on the first captured image, the display screen to move the marker image from the first position to a third position in a second corner region of the display screen, wherein the second corner region is different from the first corner region; control the imaging unit of the mobile terminal to acquire a third captured image of the eye of the user while the light source irradiates the eye of the user and the marker image is displayed at the third position; determine whether third acquisition of the optical axis vector succeeds based on the third captured image; control, based on the determination that the second acquisition of the optical axis vector succeeds based on the second captured image, the display screen to move the marker image from the second position to a fourth position of the display screen in the second corner region, wherein the fourth position is closer to the center of the display screen than the third position; control the imaging unit to acquire a fourth captured image of the eye of the user while the light source irradiates the eye of the user and the marker image is displayed at the fourth position; determine whether fourth acquisition of the optical axis vector succeeds based on the fourth captured image; and calibrate the gaze detection based on a successful combination of the first acquisition and the third acquisition of the optical axis vector or a successful combination of the second acquisition and the fourth acquisition of the optical axis vector.
 3. The information processing apparatus according to claim 2, wherein the first corner region is opposite to the second corner region.
 4. The information processing apparatus according to claim 1, wherein the circuitry is configured to control, based on the determination that the first acquisition of the optical axis vector fails based on the first captured image, the display screen to move the marker image from the first position in a horizontal direction of the display screen.
 5. The information processing apparatus according to claim 1, wherein the circuitry is configured to: compare the optical axis vector with a marker vector that connects a position of the marker image with a position of the eye; and determine that each of the first acquisition and the second acquisition of the optical axis vector succeeds based on a result of the comparison that the optical axis vector and the marker vector have a predetermined relationship.
 6. The information processing apparatus according to claim 1, wherein the circuitry is configured to: control, based on the determination that the first acquisition of the optical axis vector succeeds based on the first captured image, the display screen to move the marker image from the first position to a third position in a second corner region of the display screen, wherein the second corner region is different from the first corner region; and decrease a movement speed of the marker image as the marker image approaches the third position.
 7. The information processing apparatus according to claim 6, wherein the first corner region is opposite to the second corner region.
 8. The information processing apparatus according to claim 1, wherein the mobile terminal is a head-mounted display.
 9. An information processing method, comprising: controlling a display screen of a mobile terminal to display a marker image at a first position in a corner region of the display screen; controlling a light source of the mobile terminal to irradiate an eye of a user with infrared light; controlling an imaging unit of the mobile terminal to acquire a first captured image of the eye of the user while the light source irradiates the eye of the user and the marker image is displayed at the first position; determining whether first acquisition of an optical axis vector, which corresponds to a gaze direction of the user by a pupil-corneal reflection method, succeeds based on the first captured image; controlling, based on the determination that the first acquisition of the optical axis vector fails based on the first captured image, the display screen to move the marker image from the first position to a second position in the corner region, wherein the second position is closer to a center of the display screen than the first position; controlling the imaging unit to acquire a second captured image of the eye of the user while the light source irradiates the eye of the user and the marker image is displayed at the second position; determining whether second acquisition of the optical axis vector succeeds based on the second captured image; and calibrating gaze detection, which is performed by the mobile terminal, based on the determination that the second acquisition of the optical axis vector succeeds based on the second captured image.
 10. A non-transitory computer-readable medium having stored thereon, computer-executable instructions, which when executed by a processor of an information processing apparatus, cause the processor to execute operations, the operations comprising: controlling a display screen of a mobile terminal to display a marker image at a first position in a corner region of the display screen; control a light source of the mobile terminal to irradiate an eye of a user with infrared light; controlling an imaging unit of the mobile terminal to acquire a first captured image of the eye of the user while the light source irradiates the eye of the user and the marker image is displayed at the first position; determining whether first acquisition of an optical axis vector, which corresponds to a gaze direction of the user by a pupil-corneal reflection method, succeeds based on the first captured image; controlling, based on the determination that the first acquisition of the optical axis vector fails based on the first captured image, the display screen to move the marker image from the first position to a second position in the corner region, wherein the second position is closer to a center of the display screen than the first position; controlling the imaging unit to acquire a second captured image of the eye of the user while the light source irradiates the eye of the user and the marker image is displayed at the second position; determining whether second acquisition of the optical axis vector succeeds based on the second captured image; and calibrating gaze detection, which is performed by the mobile terminal, based on the determination that the second acquisition of the optical axis vector succeeds based on the second captured image. 